Ambari

您所在的位置：网站首页 › ambari部署hadoop集群 centos76 › Ambari

Ambari

2023-11-04 11:12| 来源: 网络整理| 查看: 265

一、服务器集群规划与安装

建议Ambari-Server单独安装在一台配置较低服务器，方便用于管理，且他不作为hadoop集群中的一台，这样有利用管理，是否配置FQDN等信息。如果FQDN域名信息为bigdata.local则各个机器名为hadoop79.bigdata.local，可以不配置，如果配置在/etc/hosts中配置。

服务器IP

功能规划（）

描述说明

10.99.67.79

（hadoop79）Ambari-Server,Mysql-5.7

如果可以考虑这台服务器单独出hadoop集群

10.99.67.83

（hadoop83）Hadoop,Yarn,Zookeeper

10.99.67.84

（hadoop84）Hadoop,Yarn,Zookeeper

10.99.67.85

（hadoop85）Hadoop,Yarn,Zookeeper

二、环境准备与安装相关软件1.mysql-5.7.20-1.el7.x86_64.rpm-bundle.tar2.mysql-connector-java-5.1.47.jar3.jdk1.8.0_231.tar.gz4.ambari-2.7.5.0-centos7.tar.gz5.HDP-3.1.5.0-centos7-rpm.tar.gz6.HDP-GPL-3.1.5.0-centos7-gpl.tar.gz7.HDP-UTILS-1.1.0.22-centos7.tar.gz8.libtirpc-0.2.4-0.16.el7.x86_64.rpm9.libtirpc-devel-0.2.4-0.16.el7.x86_64.rpm安装一些系统所有必要的工具（所有节点）yum -y install gcc gcc-c++ kernel-devel libtirpc-devel yum -y install rpm-build yum -y install net-tools yum -y install openssl openssl-devel patch yum -y install wget curl yum -y install bash-completion yum -y install unzip zip yum -y install libtirpc libtirpc-devel （如果他安装没有，可以是提供的rpm来安装） rpm -ivh libtirpc-0.2.4-0.16.el7.x86_64.rpm rpm -ivh libtirpc-devel-0.2.4-0.16.el7.x86_64.rpm关闭selinux（需重启）（所有节点）

vim /etc/sysconfig/selinux

修改SELINUX=disabled

查看状态：getenforce，返回Disable为关闭。

关闭防火墙（所有节点）systemctl stop firewalld systemctl status firewalld systemctl disable firewalld

关闭防火墙：systemctl stop firewalld

查看防火墙的状态：systemctl status firewalld

设置开机不启动：systemctl disable firewalld

查看防火墙服务是否设置开机启动：systemctl is-enabled firewalld

设置最大文件打开数最大进程数（所有节点）

# 编辑配置文件：/etc/security/limits.conf，

vim /etc/security/limits.conf

添加如下配置：

* soft nproc 204800* hard nproc 204800* soft nofile 655350* hard nofile 655350修改Linux系统内核内存参数（属于调优，可以不用调整）（所有节点）

# 编辑配置文件：/etc/sysctl.conf，

vim /etc/sysctl.conf

添加如下配置：

vm.swappiness=1vm.min_free_kbytes=4194304vm.zone_reclaim_mode=0

# 使修改生效：sysctl -p

关闭透明大页THP（需重启）（所有节点）

# 查看透明大页情况：

cat /sys/kernel/mm/transparent_hugepage/defrag cat /sys/kernel/mm/transparent_hugepage/enabled

# 关闭透明大页：vim /etc/rc.d/rc.local，添加如下配置：

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabledfiif test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defragfi

# 保存退出，然后赋予rc.local文件执行权限：chmod +x /etc/rc.d/rc.local

安装时间同步工具（所有节点）

#安装时间同步工具（所有节点）

#ntp或者chrony

yum install -y ntp systemctl enable ntpdjava安装（所有节点）

1.java解压安装

cd /opt tar -zxvf jdk1.8.0_231.tar.gz

2.编辑环境变量配置

vim /etc/profile

export JAVA_HOME=/opt/jdk1.8.0_231 export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

3.生效环境变量配置

source /etc/profile

4.检验JDK配置结果

jave -version

配置hosts（所有节点）

配置hosts

vim /etc/hosts

添加如下配置

10.99.67.79 hadoop79 10.99.67.83 hadoop83 10.99.67.84 hadoop84 10.99.67.85 hadoop85配置SSH无密码登陆（ambari-server节点或主节点，建议集群所有集群都做SSH无密码登陆）ssh-keygen -t rsa ssh-copy-id hadoop79 ssh-copy-id hadoop83 ssh-copy-id hadoop84 ssh-copy-id hadoop85

验证：

ssh hadoop79 ssh hadoop83 ssh hadoop84 ssh hadoop85

三、安装mysql解压mysql-5.7的RPM安装包

tar -xvf mysql-5.7.20-1.el7.x86_64.rpm-bundle.tar

查找系统自带mysql或者maradbrpm -qa |grep mysql rpm -qa |grep mariadb

如果找到按下面命令删除对应包

rpm -e mariadb-libs --nodeps

先yum安装部分依赖包yum -y install perl perl-devel autoconf libaio libaio-devel perl-Time-HiRes perl-JSON perl-Data-Dumper net-tools numactl安装rpm包rpm -ivh mysql-community-common-5.7.20-1.el7.x86_64.rpm rpm -ivh mysql-community-libs-5.7.20-1.el7.x86_64.rpm rpm -ivh mysql-community-libs-compat-5.7.20-1.el7.x86_64.rpm rpm -ivh mysql-community-embedded-5.7.20-1.el7.x86_64.rpm rpm -ivh mysql-community-client-5.7.20-1.el7.x86_64.rpm rpm -ivh mysql-community-devel-5.7.20-1.el7.x86_64.rpm rpm -ivh mysql-community-embedded-compat-5.7.20-1.el7.x86_64.rpm rpm -ivh mysql-community-embedded-devel-5.7.20-1.el7.x86_64.rpm rpm -ivh mysql-community-server-5.7.20-1.el7.x86_64.rpm rpm -ivh mysql-community-test-5.7.20-1.el7.x86_64.rpm查看状态并启动mysqld

# 查看mysql服务

service mysqld status #systemctl status mysqld.service

# 启动mysql服务

service mysqld start #systemctl start mysqld.service

# 查看密码

cat /var/log/mysqld.log | grep password登陆修改且密码mysql -u root -p

# 首次登录必须先修改密码（强密码）

alter user 'root'@'localhost' identified by 'root__123AbCdHjKl';

设置可以配置弱密码，（这步可以不用配置，如果使用弱密码则需要配置）

SHOW variables LIKE 'validate_password%'; set global validate_password_policy=0; set global validate_password_length=4;# 可以修改为弱密码啦 #ALTER USER 'root'@'localhost' IDENTIFIED BY "bigdata123"; #flush privileges; # 更新权限配置root可以远程访问use mysql; update user set host = '%' where user ='root'; # 配置可远程登录 flush privileges; # 更新权限 exit; # 退出配置mysql服务器编码

# 查看mysql服务的编码

show variables like '%char%';

# 修改MySQL配置文件：/etc/my.cnf

vim /etc/my.cnf

# 在[mysqld]下添加

[mysqld] character-set-client-handshake = FALSE character-set-server = utf8mb4 collation-server = utf8mb4_unicode_ci default-storage-engine=INNODB #init_connect='SET NAMES utf8mb4'

# 在[client]下添加（如果没有[client]，则创建）

[client] default-character-set = utf8mb4 [mysql] default-character-set = utf8mb4重启mysql服务service mysqld restart #systemctl restart mysqld设置开机启动chkconfig mysqld on #systemctl enable mysqld

msyql服务器安装完成。

四、安装Ambari-Server

安装一些yum工具命令，一般系统已经安装

yum install yum-utils createrepo yum-plugin-priorities安装httpd

# 安装服务

yum install httpd -y

# 启动httpd服务：systemctl start httpd

# 查看httpd状态：systemctl status httpd

# 设置开启启动：systemctl enable httpd

# 查看是否开机启动：systemctl is-enabled httpd

把HDP文件放到httpd去# 解压Ambari压缩包tar -zxvf /root/ambari-2.7.5.0-centos7.tar.gz -C /var/www/html/# 解压HDP压缩包tar -zxvf /root/HDP-3.1.5.0-centos7-rpm.tar.gz -C /var/www/html/# 解压HDP-UTILS压缩包tar -zxvf /root/HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/# 解压HDP-GPL压缩包tar -zxvf /root/HDP-GPL-3.1.5.0-centos7-gpl.tar.gz -C /var/www/html/创建对应的HDP源

在/etc/yum.repo.d/下创建4个文件，请按照实际的ambari-server服务器IP来修改下面文件内容

# ambari库源# cat ambari.repo [ambari-2.7.5.0-72]name=ambari-2.7.5.0-72baseurl=http://10.99.67.79/ambari/centos7/2.7.5.0-72/path=/enabled=1gpgkey= http://10.99.67.79/ambari/centos7/2.7.5.0-72/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsgpgcheck=1priority=1# HDP库源# cat HDP.repo [HDP-3.1.5]name=HDP-3.1.5baseurl=http://10.99.67.79/HDP/centos7/3.1.5.0-152/path=/enabled=1gpgkey=http://10.99.67.70/HDP/centos7/3.1.5.0-152/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsgpgcheck=1priority=1# HDP-UTILS库源# cat HDP-UTILS.repo [HDP-UTILS-1.1.0.22]name=HDP-UTILS-1.1.0.22baseurl=http://10.99.67.79/HDP-UTILS/centos7/1.1.0.22/path=/enabled=1gpgkey=http://10.99.67.79/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsgpgcheck=1priority=1# HDP-GPL库源# cat HDP-GPL.repo [HDP-3.1.5-GPL]name=HDP-3.1.5-GPLbaseurl=http://10.99.67.79/HDP-GPL/centos7/3.1.5.0-152/path=/enabled=1gpgkey=http://10.99.67.79/HDP-GPL/centos7/3.1.5.0-152/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsgpgcheck=1priority=1拷贝HDP库源文件到所有服务器scp *.repo root@hadoop83:/etc/yum.repos.d/ scp *.repo root@hadoop84:/etc/yum.repos.d/ scp *.repo root@hadoop85:/etc/yum.repos.d/

测试HDP源（所有服务器）yum clean all yum repolist yum makecache安装ambari-serveryum -y install ambari-server安装ambari-server的mysql的驱动mkdir /usr/share/java cp mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-server/resources/配置ambari-server的mysql的驱动配置/etc/ambari-server/conf/ambari.properties，添加如下行 server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar执行ambari-server的mysql的驱动注册ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jarUsing python /usr/bin/pythonSetup ambari-serverCopying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources/mysql-connector-java.jarIf you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.JDBC driver was successfully initialized.Ambari Server 'setup' completed successfully.初始化ambari-serverambari-server setup# ambari-server setupUsing python /usr/bin/pythonSetup ambari-serverChecking SELinux...SELinux status is 'disabled'Customize user account for ambari-server daemon [y/n] (n)? yEnter user account for ambari-server daemon (root):Adjusting ambari-server permissions and ownership...Checking firewall status...Checking JDK...[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8[2] Custom JDK==============================================================================Enter choice (1): 2WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.Path to JAVA_HOME: /opt/jdk1.8.0_231Validating JDK on Ambari Server...done.Check JDK version for Ambari Server...JDK version found: 8Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.Checking GPL software agreement...GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.htmlEnable Ambari Server to download and install GPL Licensed LZO packages [y/n] (y)? Completing setup...Configuring database...Enter advanced database configuration [y/n] (n)? yConfiguring database...==============================================================================Choose one of the following options:[1] - PostgreSQL (Embedded)[2] - Oracle[3] - MySQL / MariaDB[4] - PostgreSQL[5] - Microsoft SQL Server (Tech Preview)[6] - SQL Anywhere[7] - BDB==============================================================================Enter choice (1): 3Hostname (localhost): Port (3306): Database name (ambari): Username (ambari): Enter Database Password (bigdata): ambari123Re-enter password: Configuring ambari database...Configuring remote database connection properties...WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sqlProceed with configuring remote database connection properties [y/n] (y)? yExtracting system views...ambari-admin-2.7.5.0.72.jar....Ambari repo file doesn't contain latest json url, skipping repoinfos modificationAdjusting ambari-server permissions and ownership...Ambari Server 'setup' completed successfully.ambari数据库的导入-- 创建ambari数据库CREATE DATABASE ambari; -- 创建ambari用户CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari123'; -- 给创建的ambari赋权GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';

#CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'ambari123'; #GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost'; #CREATE USER 'ambari'@'hadoop79' IDENTIFIED BY 'ambari123'; #GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'hadoop79';-- 刷新权限flush privileges;

-- 导入ambari的数据库use ambari;source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql-- 验证数据表show tables;-- 验证用户use mysql; select host,user from user where user='ambari';-- #创建其它大数据组件的库CREATE DATABASE hive; use hive; CREATE USER 'hive'@'%' IDENTIFIED BY 'hive123'; GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%'; #CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive123'; #GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost'; #CREATE USER 'hive'@'hadoop79' IDENTIFIED BY 'hive123'; #GRANT ALL PRIVILEGES ON *.* TO 'hive'@'hadoop79'; FLUSH PRIVILEGES;

CREATE DATABASE oozie; use oozie; CREATE USER 'oozie'@'%' IDENTIFIED BY 'oozie123'; GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%'; #CREATE USER 'oozie'@'localhost' IDENTIFIED BY 'oozie123'; #GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'localhost'; #CREATE USER 'oozie'@'hadoop79' IDENTIFIED BY 'oozie123'; #GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'hadoop79'; FLUSH PRIVILEGES;

安装ambari-agent(所有节点执行)yum -y install ambari-agent systemctl start ambari-agent启动ambari服务启动ambari服务#启动ambari-server start#查看状态#ambari-server status#停止ambari-server stop页面创建集群

打开http://10.99.67.79:8080 创建集群。

五、通过页面创建hadoop集群

打开http://10.99.67.79:8080 ，输入admin/admin登陆。

Ambari-2.7.5+HDP-3.1.5安装手册部署1_ambari